Research Methods 2022
Ethan Milne // Ivey Business School
September 12, 2022
Gameplan:
Why webscrape?
Scraping with APIs
Scraping without APIs
Introduction
Why Webscrape?
What is Webscraping?
Why Webscrape?
Scraping with APIs
What is an API?
Application Program Interface
Scraping with APIs
APIs: Pros and Cons
Pros |
Cons |
| Easy to use | Not available for every site |
| Well-documented | No control over data available |
| Limited liability | Usage restrictions |
Scraping with APIs
Twitter Study
Twitter Study: Context
Twitter Study: Methods
Twitter Study: Methods
Twitter Study: Methods
library(academictwitteR)
bk_tweets <- get_all_tweets(
query = "@BurgerKingUK",
start_tweets = "2021-03-06T00:00:00Z",
end_tweets = "2021-03-13T00:00:00Z",
n = 10000000
)
for (i in unique_authors) {
past_tweets[[i]] <- get_all_tweets(
users = unique_authors$username[start:end],
start_tweets = "2021-03-01T00:00:01.000Z",
end_tweets = "2021-03-08T00:00:01.000Z",
n = 1000000
)
}Twitter Study: Methods
Twitter Study: Methods
Results
| Model Summaries | ||||||
|---|---|---|---|---|---|---|
| Characteristic | Without Present Tweets | With Present Tweets | ||||
| IRR1 | 95% CI1 | p-value | IRR1 | 95% CI1 | p-value | |
| NormalQuantity (past) | 0.70 | 0.68, 0.73 | <0.001 | 0.74 | 0.71, 0.76 | <0.001 |
| NormalLikes (past) | 0.96 | 0.93, 0.98 | <0.001 | 0.93 | 0.91, 0.96 | <0.001 |
| OutrageQuantity (past) | 1.72 | 1.64, 1.81 | <0.001 | 1.68 | 1.61, 1.76 | <0.001 |
| OutrageLikes (past) | 1.12 | 1.09, 1.15 | <0.001 | 1.09 | 1.06, 1.13 | <0.001 |
| Followers | 1.00 | 1.00, 1.00 | 0.4 | 1.00 | 1.00, 1.00 | >0.9 |
| TotalTweets | 1.00 | 1.00, 1.00 | <0.001 | 1.00 | 1.00, 1.00 | <0.001 |
| NormalQuantity (present) | 4.08 | 3.95, 4.21 | <0.001 | |||
| 1 IRR = Incidence Rate Ratio, CI = Confidence Interval | ||||||
Twitter Study: Results
Scraping Without APIs
Building scrapers is hard
No prebuilt code
Websites not designed for scraping
Websites protected against scraping
Scraping without APIs
Building scrapers is rewarding
You know your data inside and out
Your data is unique
Your scraper is a contribution
Scraping Without APIs
Scraping Without APIs
Fanfiction Study
Fanfiction Study: Methods
Fanfiction Study: Methods
Fanfiction Study: Methods
Fanfiction Study: Methods
Fanfiction Study: Methods
Get HTML pages
Fanfiction Study: Scraper
Get HTML
get_html(input = "https://archiveofourown.org/tags/Fantastic%20Beasts%20and%20Where%20to%20Find%20Them%20(Movies)/works"){html_document}
<html lang="en">
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body class="logged-out">\n <div id="outer" class="wrapper">\n <u ...Fanfiction Study: Scraper
Read HTML
get_html(input = "https://archiveofourown.org/tags/Fantastic%20Beasts%20and%20Where%20to%20Find%20Them%20(Movies)/works") |>
html_text2()[1] "Main Content\n\nWhile we've done our best to make the core functionality of this site accessible without javascript, it will work better with it enabled. Please consider turning it on!\n\nArchive of Our Own beta(function(){for(var g=\"function\"==typeof Object.defineProperties?Object.defineProperty:function(b,c,a){if(a.get||a.set)throw new TypeError(\"ES3 does not support getters and setters.\");b!=Array.prototype&&b!=Object.prototype&&(b[c]=a.value)},h=\"undefined\"!=typeof window&&window===this?this:\"undefined\"!=typeof global&&null!=global?global:this,k=[\"String\",\"prototype\",\"repeat\"],l=0;l<k.length-1;l++){var m=k[l];m in h||(h[m]={});h=h[m]}var n=k[k.length-1],p=h[n],q=p?p:function(b){var c;if(null==this)throw new TypeError(\"The 'this' value for String.prototype.repeat must not be null or undefined\");c=this+\"\";if(0>b||1342177279<b)throw new RangeError(\"Invalid count value\");b|=0;for(var a=\"\";b;)if(b&1&&(a+=c),b>>>=1)c+=c;return a};q!=p&&null!=q&&g(h,n,{configurable:!0,writable:!0,value:q});var t=this;function u(b,c){var a=b.split(\".\"),d=t;a[0]in d||!d.execScript||d.execScript(\"var \"+a[0]);for(var e;a.length&&(e=a.shift());)a.length||void 0===c?d[e]?d=d[e]:d=d[e]={}:d[e]=c};function v(b){var c=b.length;if(0<c){for(var a=Array(c),d=0;d<c;d++)a[d]=b[d];return a}return[]};function w(b){var c=window;if(c.addEventListener)c.addEventListener(\"load\",b,!1);else if(c.attachEvent)c.attachEvent(\"onload\",b);else{var a=c.onload;c.onload=function(){b.call(this);a&&a.call(this)}}};var x;function y(b,c,a,d,e){this.h=b;this.j=c;this.l=a;this.f=e;this.g={height:window.innerHeight||document.documentElement.clientHeight||document.body.clientHeight,width:window.innerWidth||document.documentElement.clientWidth||document.body.clientWidth};this.i=d;this.b={};this.a=[];this.c={}}function z(b,c){var a,d,e=c.getAttribute(\"data-pagespeed-url-hash\");if(a=e&&!(e in b.c))if(0>=c.offsetWidth&&0>=c.offsetHeight)a=!1;else{d=c.getBoundingClientRect();var f=document.body;a=d.top+(\"pageYOffset\"in window?window.pageYOffset:(document.documentElement||f.parentNode||f).scrollTop);d=d.left+(\"pageXOffset\"in window?window.pageXOffset:(document.documentElement||f.parentNode||f).scrollLeft);f=a.toString()+\",\"+d;b.b.hasOwnProperty(f)?a=!1:(b.b[f]=!0,a=a<=b.g.height&&d<=b.g.width)}a&&(b.a.push(e),b.c[e]=!0)}y.prototype.checkImageForCriticality=function(b){b.getBoundingClientRect&&z(this,b)};u(\"pagespeed.CriticalImages.checkImageForCriticality\",function(b){x.checkImageForCriticality(b)});u(\"pagespeed.CriticalImages.checkCriticalImages\",function(){A(x)});function A(b){b.b={};for(var c=[\"IMG\",\"INPUT\"],a=[],d=0;d<c.length;++d)a=a.concat(v(document.getElementsByTagName(c[d])));if(a.length&&a[0].getBoundingClientRect){for(d=0;c=a[d];++d)z(b,c);a=\"oh=\"+b.l;b.f&&(a+=\"&n=\"+b.f);if(c=!!b.a.length)for(a+=\"&ci=\"+encodeURIComponent(b.a[0]),d=1;d<b.a.length;++d){var e=\",\"+encodeURIComponent(b.a[d]);131072>=a.length+e.length&&(a+=e)}b.i&&(e=\"&rd=\"+encodeURIComponent(JSON.stringify(B())),131072>=a.length+e.length&&(a+=e),c=!0);C=a;if(c){d=b.h;b=b.j;var f;if(window.XMLHttpRequest)f=new XMLHttpRequest;else if(window.ActiveXObject)try{f=new ActiveXObject(\"Msxml2.XMLHTTP\")}catch(r){try{f=new ActiveXObject(\"Microsoft.XMLHTTP\")}catch(D){}}f&&(f.open(\"POST\",d+(-1==d.indexOf(\"?\")?\"?\":\"&\")+\"url=\"+encodeURIComponent(b)),f.setRequestHeader(\"Content-Type\",\"application/x-www-form-urlencoded\"),f.send(a))}}}function B(){var b={},c;c=document.getElementsByTagName(\"IMG\");if(!c.length)return{};var a=c[0];if(!(\"naturalWidth\"in a&&\"naturalHeight\"in a))return{};for(var d=0;a=c[d];++d){var e=a.getAttribute(\"data-pagespeed-url-hash\");e&&(!(e in b)&&0<a.width&&0<a.height&&0<a.naturalWidth&&0<a.naturalHeight||e in b&&a.width>=b[e].o&&a.height>=b[e].m)&&(b[e]={rw:a.width,rh:a.height,ow:a.naturalWidth,oh:a.naturalHeight})}return b}var C=\"\";u(\"pagespeed.CriticalImages.getBeaconData\",function(){return C});u(\"pagespeed.CriticalImages.Run\",function(b,c,a,d,e,f){var r=new y(b,c,a,e,f);x=r;d&&w(function(){window.setTimeout(function(){A(r)},0)})});})();pagespeed.CriticalImages.Run('/ngx_pagespeed_beacon','http://archiveofourown.org/tags/Fantastic%20Beasts%20and%20Where%20to%20Find%20Them%20(Movies)/works','eVfAzEuqgZ',true,false,'gEwencUN6h0');\n\nLog In\n\nUser name or email:\nPassword:\n\nRemember Me\n\nForgot password?\nGet an Invitation\nSite Navigation\nFandoms\nAll Fandoms\nAnime & Manga\nBooks & Literature\nCartoons & Comics & Graphic Novels\nCelebrities & Real People\nMovies\nMusic & Bands\nOther Media\nTheater\nTV Shows\nVideo Games\nUncategorized Fandoms\nBrowse\nWorks\nBookmarks\nTags\nCollections\nSearch\nWorks\nBookmarks\nTags\nPeople\nAbout\nAbout Us\nNews\nFAQ\nWrangling Guidelines\nDonate or Volunteer\nSearch Works\n\nWork Search: tip: austen words:10000-50000 sort:title\n\n1 - 20 of 15646 Works in Fantastic Beasts and Where to Find Them (Movies)\nNavigation and Actions\nWorks\nBookmarks\nFilters\nRSS Feed\nPages Navigation\n← Previous\n1\n2\n3\n4\n5\n6\n7\n8\n9\n…\n782\n783\nNext →\nListing Works\n神奇动物在哥谭 by do not cache Bucket_Cookies\nFandoms: Batman - All Media Types, Batman: Under the Red Hood (2010), Fantastic Beasts and Where to Find Them (Movies), Harry Potter - J. K. Rowling \nGeneral Audiences\nNo Archive Warnings Apply\nGen\nWork in Progress\n\n12 Sep 2022\n\nTags\nNo Archive Warnings Apply\nTim Drake & Dick Grayson & Jason Todd & Bruce Wayne & Damian Wayne\nAlfred Pennyworth & Jason Todd\nTim Drake & Dick Grayson & Alfred Pennyworth & Jason Todd & Bruce Wayne & Damian Wayne\nBatfamily Members & Jason Todd\nJason Todd\nBruce Wayne\nDick Grayson\nTim Drake\nDamian Wayne\nAlfred Pennyworth\nNewt Scamander\nRoman Sionis\nCrossover\nSummary\n\n关于著名神奇动物学家亲传弟子红头罩与麻瓜蝙蝠一家在哥谭打击走私神奇动物黑恶势力的故事\n\nSeries\nPart 1 of 神奇动物学家红头罩\nLanguage:\n中文-普通话 國語\nWords:\n5,900\nChapters:\n3/?\nKudos:\n4\nHits:\n79\nForgive Me by do not cache Cainz\nFandoms: Fantastic Beasts and Where to Find Them (Movies), Harry Potter - J. K. Rowling \nNot Rated\nNo Archive Warnings Apply\nNo category\nWork in Progress\n\n12 Sep 2022\n\nTags\nNo Archive Warnings Apply\nAlbus Dumbledore\nAberforth Dumbledore\nAurelius Dumbledore\nCredence Barebone\nCredence Barebone's Obscurus\nCredence Barebone Needs a Hug\nCredence Barebone Gets a Hug\nCredence Barebone Deserves Better\nObscurial Credence Barebone\nCredence Barebone Heals\nCredence Barebone Lives\nGood Albus Dumbledore\nYoung Albus Dumbledore\nAberforth being a good father\nFather-Son Relationship\nFatherhood\nPast Abuse\nAbuse\nChild Abuse\nPast Child Abuse\nGellert Grindelwald Being an Asshole\nMovie 3: Fantastic Beasts: The Secrets of Dumbledore\nPost-Movie 3: Fantastic Beasts: The Secrets of Dumbledore\nFamily Bonding\nFamily Feels\nSummary\n\n“He still could not fathom that the man he was sent to kill had not just spared his pathetic life, but accepted him into his family. After everything he had done, after all the harm he had caused, both Dumbledores had welcomed him with open arms.”\n\nor\n\nAurelius tries to adjust to normal life, but old fears come back to haunt him.\n\nLanguage:\nEnglish\nWords:\n8,148\nChapters:\n6/10\nComments:\n28\nKudos:\n192\nBookmarks:\n28\nHits:\n2704\nMagical Marvel (HP X MCU) SI by do not cache athass_prkr\nFandoms: Harry Potter - J. K. Rowling, Marvel Cinematic Universe, The Avengers (Marvel Movies), The Avengers (Marvel) - All Media Types, X-Men - All Media Types, Fantastic Beasts and Where to Find Them (Movies), Fantastic Four, Spider-Man - All Media Types \nMature\nChoose Not To Use Archive Warnings\nF/F, F/M, Gen, Multi\nWork in Progress\n\n12 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nJames Potter/Lily Evans Potter\nJasmine Potter (OFC)\nLily Evans Potter\nJames Potter\nAlbus Dumbledore\nRose Potter (OFC)\nNathan Potter (OMC)\nSteve Rogers\nAncient One (Marvel)\nMerlin (Harry Potter)\nStephen Strange\nGodric Gryffindor\nSalazar Slytherin\nRowena Ravenclaw\nHelga Hufflepuff\nMorgana | Morgan le Fay (Harry Potter)\nMordred (Arthurian)\nCharles Xavier\nEn Sabah Nur\nErik Lehnsherr\nHela (Marvel)\nThor (Marvel)\nOdin (Marvel)\nTony Stark\nPeter Parker\nTom Riddle | Voldemort\nGellert Grindelwald\nNatasha Romanov (Marvel)\nWanda Maximoff\nDeath (Harry Potter)\nThe Morrigan (Marvel)\nTime Travel\nSelf-Insert\nReincarnation\nWrong Boy-Who-Lived (Harry Potter)\nAtlantis\nAncient Greece\nCamelot\nHogwarts Founders Era\nAncient Egypt\nImmortality\nMaster of Death (Harry Potter)\nJasmine in history\nLegends\nHarry Potter in MCU\nMutant Powers\nWadjet\nMedusa - Freeform\nReferences to Ancient Greek Religion & Lore\nSummary\n\n[A SI Harry Potter and Marvel (MCU) Xover Fanfiction]\n\n---------------------------------------------------------------\n\nA sick 16 year old MC dies. At the same time, emerald eyes open up confused. Shouldn't I be dead? Why the hell am I so tiny? What's her name? What are these voices in my head?\nFollow the young telepath as they make way in a world of gods and magic. Will they make their mark in this scary world or did they already do so. F*ck, time travel is really confusing.\n\n----------------------------------------------------------------\nI know that the beginning is not that good, tbh. It gets better with time, believe it.\n\nI don't own pretty much anything in this Fanfic.\n\nI took a lot of inspiration from the fic \"A Long Journey Home\" and used the prologue from \"Rise of Arcane\" to get things started.\n\n\nLanguage:\nEnglish\nWords:\n233,591\nChapters:\n100/?\nComments:\n20\nKudos:\n213\nBookmarks:\n74\nHits:\n6781\nThe Rise of The True Lord by do not cache AllyLitteWolf, Lucifer616\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nExplicit\nGraphic Depictions Of Violence\nM/M, Multi\nWork in Progress\n\n12 Sep 2022\n\nTags\nGraphic Depictions Of Violence\nDraco Malfoy/Harry Potter\nDraco Malfoy & Harry Potter\nHarry Potter/Tom Riddle\nHarry Potter/Tom Riddle | Voldemort\nHarry Potter/Fred Weasley/George Weasley\nHarry Potter/Fred Weasley\nHarry Potter & Fred Weasley & George Weasley\nHarry Potter & Fred Weasley\nHarry Potter/George Weasley\nHarry Potter & George Weasley\nNeville Longbottom/Harry Potter\nNeville Longbottom & Harry Potter\nColin Creevey/Harry Potter\nColin Creevey & Harry Potter\nHarry Potter\nTom Riddle | Voldemort\nVoldemort (Harry Potter)\nDraco Malfoy\nLucius Malfoy\nNarcissa Black Malfoy\nAlbus Dumbledore\nMinerva McGonagall\nBlaise Zabini\nTheodore Nott\nPansy Parkinson\nOriginal Characters\nGoblins (Harry Potter)\nPetunia\nColin Creevey\nNeville Longbottom\nFred Weasley\nGeorge Weasley\nRomance\nDrama\nFantasy\nEventual Smut\nHarems\nYaoi\nPolyamorous Character\nAlbus Dumbledore Bashing\nMinerva McGonagall Bashing\nDark\nOther Additional Tags to Be Added\nHarryxAll\nHarem\nHarrybowl\nSummary\n\nUm jovem destinado ao topo do mundo. Mas mantido aprisionado no local mais profundo por aqueles que o temem ou que desejam usá-lo. Finalmente quebrou as correntes que o prendiam e se libertou para o mundo, para mostra-los o que o verdadeiro senhor é capaz.\n\nO mau trará consigo a morte\nVinda de um falso Deus\nA criança nascida dos lobos\nDotada de um poder inimaginável\n\nNa 17º lua da cria dos lobos\nO Deus Verdadeiro irá triunfar,\nsobre aquele que tinha a boca cheia de mentiras\nEscondido por trás de um sorriso carinhoso o Diabo se esconde\nO verdadeiro mau está escondido dentro da falsa luz\nO filhote da serpente\nCrescendo maltratado,\nserá mil vezes mais cruel quando mais velho ficar\n\nLanguage:\nPortuguês brasileiro\nWords:\n206,311\nChapters:\n72/?\nComments:\n69\nKudos:\n157\nBookmarks:\n20\nHits:\n7736\nBefore the Past by do not cache Justaferalgoblingal\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nNot Rated\nChoose Not To Use Archive Warnings\nM/M, Multi\nWork in Progress\n\n12 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nOriginal Percival Graves/Credence Barebone/Newt Scamander/Harry Potter\nCredence Barebone/Original Percival Graves\nCredence Barebone/Newt Scamander\nCredence Barebone/Harry Potter\nOriginal Percival Graves/Newt Scamander\nOriginal Percival Graves/Harry Potter\nHarry Potter/Newt Scamander\nNewt Scamander\nHarry Potter\nTeddy Lupin\nCredence Barebone\nOriginal Percival Graves\nModesty Barebone\nMary Lou Barebone\nChastity Barebone\nTina Goldstein\nQueenie Goldstein\nJacob Kowalski\nTom Riddle | Voldemort\nMerope Gaunt\nOther Character Tags to Be Added\nOriginal Goblin Characters\nRamgok\nOriginal Male Character(s)\nOriginal Goblin Character(s)\nNaemma\nSlow Burn\nMaster of Death Harry Potter\nCanonical Child Abuse\nWarning: Mary Lou Barebone\nshe's a bitch\nCredence Barebone Needs a Hug\nCredence Barebone Gets a Hug\nCredence Barebone Learning Magic\nI Wrote This Instead of Sleeping\nNo beta: we die like men\nPercival Graves does not die\nHe is too hot\ntime traveler harry potter\nTeddy Lupin is a sass monster\nWe love him\nHarry Potter Epilogue What Epilogue | EWE\nParental Harry Potter\nOther Additional Tags to Be Added\nCanonical Character Death\nImplied/Referenced Rape/Non-con\nidk if it is but imma tag it anyway just incase\nChildbirth\nDeath in Childbirth\nRituals\nidek man\nlike so much happens in chpt 4\nSummary\n\nExerpt:\n\"The only thing that Harry could remember before hitting the cold, hard, cobblestone of the street was a feeling not unlike a portkey and a whirlwind of color before blacking out. Something was wrong. Something had happened.\"\n\nWhen suddenly pulled into the past with his son, Teddy, Harry Potter has to figure out exactly what he wants and why he suddenly has words all over his body.\n\nDeciding that he can only make the world better Harry sets out on a path of creating a better wizarding world. After all you can't fuck it up worse than it already is. Right?\n\nLanguage:\nEnglish\nWords:\n11,077\nChapters:\n7/?\nComments:\n39\nKudos:\n307\nBookmarks:\n100\nHits:\n3868\nMysteries That Spread (When The Alliance Takes Over) by do not cache The_bookworm_fangirl1\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nNot Rated\nNo Archive Warnings Apply\nF/M\nWork in Progress\n\n12 Sep 2022\n\nTags\nNo Archive Warnings Apply\nViktor Krum & Original Female Character(s)\nHermione Granger & Draco Malfoy\nLuna Lovegood & Rolf Scamander\nNeville Longbottom & Luna Lovegood & Harry Potter & Rolf Scamander\nOriginal Black Family Characters (Harry Potter)\nHarry Potter\nTom Riddle | Voldemort\nTom Riddle\nNeville Longbottom\nHermione Granger\nSirius Black\nOriginal Male Character(s)\nOriginal Potter Family Characters (Harry Potter)\nDaphne Greengrass\nFred Weasley\nGinny Weasley\nLuna Lovegood\nGeorge Weasley\nRemus Lupin\nAlbus Dumbledore\nRegulus Black\nTheodore Nott\nBlaise Zabini\nCho Chang\nKatie Bell\nRolf Scamander\nCedric Diggory\nSusan Bones\nPansy Parkinson\nParvati Patil\nPadma Patil\nFleur Delacour\nLavender Brown\nJustin Finch-Fletchley\nGreengrass Family (Harry Potter)\nAlternate Universe - Canon Divergence\nSirius Black Lives\nSirius Black Free from Azkaban\nPOV Original Female Character\nHarry Potter Epilogue What Epilogue | EWE\nAlbus Dumbledore Bashing\nManipulative Albus Dumbledore\nGood Tom Riddle\nDark Sirius Black\nPureblood Politics (Harry Potter)\nSummary\n\nSomeone finally decided to interfere, changing the course of Harry Potter's second year and beyond.What if Harry isn't the only one looking for the cause of petrification in 1992?\n\nWhere a group of students come together to change Hogwarts and The Wizarding World, altering Harry's world forever. Lynette Ambrosius id determined to fix her world, but she won't be able to do it on her own. In an effort to unite Hogwarts, she brings together a group of students from all houses to try to protect the student body, using the powerful rumor mill to their advantage. Hogwarts will not know what hit her, as 'The Alliance' are ready to take control.\n\nLanguage:\nEnglish\nWords:\n16,902\nChapters:\n5/11\nKudos:\n5\nBookmarks:\n3\nHits:\n139\nThe Tale of Albus Dumbledore: The Lives of Friends by do not cache mysticxy\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nTeen And Up Audiences\nNo Archive Warnings Apply\nNo category\nWork in Progress\n\n12 Sep 2022\n\nTags\nNo Archive Warnings Apply\nAlbus Dumbledore & Everyone\nHarry Potter/Ginny Weasley\nTina Goldstein/Newt Scamander\nQueenie Goldstein/Jacob Kowalski\nAlbus Dumbledore/Gellert Grindelwald\nHermione Granger/Ron Weasley\nAlbus Dumbledore\nHarry Potter\nNewt Scamander\nGinny Weasley\nTina Goldstein\nJacob Kowalski\nQueenie Goldstein\nEulalie Hicks\nTheseus Scamander\nLeta Lestrange\nHogwarts Staff\nGellert Grindelwald\nAberforth Dumbledore\nHermione Granger\nRon Weasley\nWizard's Duels (Harry Potter)\nFluff and Angst\nFather-Son Relationship\nBroken Promises\nSad Albus Dumbledore\nSummary\n\nIn here we cover povs of others, especially Harry, and how they interact with Albus and what they thought of him and his actions.\n\nSeries\nPart 3 of The Rise of Light and Darkness and a New Age\nLanguage:\nEnglish\nWords:\n10,789\nChapters:\n2/?\nKudos:\n3\nHits:\n127\nDon't Let Me Go by do not cache blaisegellert\nFandoms: Harry Potter - J. K. Rowling, Harry Potter and the Cursed Child - Thorne & Rowling, Fantastic Beasts and Where to Find Them (Movies) \nMature\nGraphic Depictions Of Violence\nM/M, F/M\nWork in Progress\n\n12 Sep 2022\n\nTags\nGraphic Depictions Of Violence\nGellert Grindelwald/Blaise Zabini\nMillicent Bulstrode/Original Male Character(s)\nGellert Grindelwald\nBlaise Zabini\nMillicent Bulstrode\nDelphi (Harry Potter)\nVinda Rosier\nOriginal Female Character(s)\nMrs Zabini (Harry Potter)\nZabini Family (Harry Potter)\nBlaise Zabini's Father\nBlaise Zabini's Mother's Seventh Husband\nLestrange Family (Harry Potter)\nHouse Elves\nTime Turners (Harry Potter)\nMurder\nPre-Movie 1: Fantastic Beasts and Where to Find Them\nMovie 2: Fantastic Beasts: The Crimes of Grindelwald\nPost-Movie 3: Fantastic Beasts: The Secrets of Dumbledore\nGellert Grindelwald Lives\nPOV Blaise Zabini\nPOV Gellert Grindelwald\nPureblood Culture (Harry Potter)\nDark Magic\nThe Dark Arts (Harry Potter)\nElder Wand (Harry Potter)\nThe Deathly Hallows\nHarry Potter Next Generation\nHarry Potter and the Cursed Child Compliant\nGrindelwald Lives\ngrindelwald deserves better\ngrindelwald wins\nSong Lyrics\nSummary\n\nWhen Gellert Grindelwald opens the door to visitors late one evening in the early nineteen-twenties his life changes forever. They claim to be friends...from the future, and they have disturbing news of his imprisonment and death. Can they change it all together or is something more sinister happening here?\n\nLanguage:\nEnglish\nWords:\n34,800\nChapters:\n17/30\nKudos:\n7\nBookmarks:\n1\nHits:\n217\nIn Another World by do not cache Tarwyn\nFandoms: Fantastic Beasts and Where to Find Them (Movies), Harry Potter - J. K. Rowling \nTeen And Up Audiences\nChoose Not To Use Archive Warnings\nM/M\nWork in Progress\n\n12 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nAlbus Dumbledore/Gellert Grindelwald\nAlbus Dumbledore & Gellert Grindelwald\nGellert Grindelwald\nAlbus Dumbledore\nOriginal Characters\nCredence Barebone\nAurelius Dumbledore\nParallel Universes\nAlternate Universe\nthere will be angst\nGellert and Albus are 'married'\nTheir relationship is complicated\nOther Additional Tags to Be Added\nI'm Bad At Titles\ntitle might change\nManipulative Gellert Grindelwald\nSummary\n\nAs the son of the two most powerful wizards of their time, 17-year-old Merlin leads a relatively quiet life. He is about to enter his 7th year at Hogwarts, when everything is going downhill. He starts to question some aspects of his own parents' past and inexplicable things start to happen. Eventually, after touching a strange piece of jewelry that plays a crucial role in his life, he finds himself in another world very similar to his own with familiar faces - except for the fact that in this reality, his own father is the mastermind behind the impending war, which is about to inevitably break out...\n\nLanguage:\nEnglish\nWords:\n1,123\nChapters:\n1/?\nComments:\n1\nKudos:\n5\nHits:\n42\nLet It Run Free by do not cache Anonymous\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nTeen And Up Audiences\nNo Archive Warnings Apply\nF/M\nComplete Work\n\n12 Sep 2022\n\nTags\nNo Archive Warnings Apply\nBunty Broadacre/Newt Scamander\nNewt Scamander\nBunty Broadacre\nNot Canon Compliant - Fantastic Beasts Movies\nPOV Newt Scamander\nAdorable Newt Scamander\nOblivious Newt Scamander\nCinnamon Roll Newt Scamander\nNewt Scamander is a Sweetheart\nTransgender Newt Scamander\nAccidental Relationship\nFalling In Love\nBoss/Employee Relationship\nFluff and Hurt/Comfort\nFluff and Mush\nMutual Pining\nDeveloping Relationship\nLGBTQ Themes\nLove Confessions\nFirst Kiss\nHappy Ending\nFriendship\nSelf-Doubt\nFriends to Lovers\nFriendship/Love\nOne Shot\nSummary\n\nNewt expected to deal with a lot in his life. Some were welcome challenges; some were predictable considering his line of work and others were a pleasant surprise. Jumping, or sliding, into things without thinking though was a first. Something he found he wasn’t exactly a fan of when it left him trying to run from the truth.\n\nLanguage:\nEnglish\nWords:\n2,138\nChapters:\n1/1\nCollections:\n1\nComments:\n2\nKudos:\n5\nHits:\n24\nPerpetuation by do not cache DrSchaf\nFandoms: Fantastic Beasts and Where to Find Them (Movies) \nMature\nNo Archive Warnings Apply\nM/M\nComplete Work\n\n11 Sep 2022\n\nTags\nNo Archive Warnings Apply\nAlbus Dumbledore/Gellert Grindelwald\nAlbus Dumbledore\nGellert Grindelwald\nUnresolved Tension\nPining\nLove Confessions\nBad Decisions\nSexual Content\nSummary\n\nHe was so foolish to think his heart could withstand having Gellert close again. There was no doubt at all—only about the time it would take until his rational thoughts were once again drowned out by love and lust.\n\nLanguage:\nEnglish\nWords:\n9,900\nChapters:\n1/1\nComments:\n3\nKudos:\n26\nBookmarks:\n6\nHits:\n222\nNewt's Inner Beasts by do not cache riiviastears\nFandoms: Fantastic Beasts and Where to Find Them (Movies) \nTeen And Up Audiences\nChoose Not To Use Archive Warnings\nM/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nNewt Scamander/Theseus Scamander\nGellert Grindelwald/Newt Scamander\nNewt Scamander\nTheseus Scamander\nNewt Scamander's Magical Beasts\nGellert Grindelwald\nTina Goldstein\nSibling Incest\nDark Theseus Scamander\nGood Gellert Grindelwald\nSummary\n\nIt wasn't as if Newt wasn't trying, though. He was. He really was. But it clearly wasn't enough for his elder brother, who wanted him to be the sweet, darling, innocent loverboy that he was before Grindlewald happened. He wanted him to become the same, sweet, innocent flowerboy whom he had fucked all day and all night. He wanted the Newt who got all flustered and cute and pink with shame when Theseus was inside him.\n\nTheseus wanted Newt, and all Newt wanted was to let go of his past, his present and his future. He wanted to end his very being. He wanted to fade away like dust particles do, riding away by the various pressures from forced or unforced breezes. It was freeing. The thought felt freeing.\n\nBut it wasn't as if Theseus would ever allow such a thing.\n\nOr, the one where Theseus is in love with Newt, but Newt has the hots for Grindelwald instead. Also, Newt is depressed.\n\nLanguage:\nEnglish\nWords:\n900\nChapters:\n1/?\nKudos:\n3\nBookmarks:\n1\nHits:\n40\nHope by do not cache ForbiddenGalaxy\nFandoms: Fantastic Beasts and Where to Find Them (Movies) \nGeneral Audiences\nChoose Not To Use Archive Warnings\nF/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nTina Goldstein/Newt Scamander\nNewt Scamander & Theseus Scamander\nQueenie Goldstein & Tina Goldstein & Jacob Kowalski & Newt Scamander\nTina Goldstein & Queenie Goldstein\nLeta Lestrange/Theseus Scamander\nLeta Lestrange & Newt Scamander\nLeta Lestrange & Tina Goldstein\nQueenie Goldstein/Jacob Kowalski\nCredence Barebone/Nagini\nNewt Scamander\nTina Goldstein\nTheseus Scamander\nLeta Lestrange\nQueenie Goldstein\nJacob Kowalski\nCredence Barebone\nNagini\nAlbus Dumbledore\nGellert Grindelwald\nVinda Rosier\nOriginal Characters\nFluff\nFound Family\nnewtina\nAngst\nNo Smut\nAU\nAlternate Storyline after Crimes of Grindelwald\nRomance\nsmall Children\nPregnancy\nSummary\n\nNot even being on the other side of the world can Newt and Tina escape the brewing the war and after a hastily written letter from Theseus they are pulled straight back into the middle of the storm. As more and more things threaten to pull their world together they have to keep fighting. If for nothing else- for the future they now share.\n\nThe third and final instalment of the Falling trilogy.\n\nSeries\nPart 3 of Falling\nLanguage:\nEnglish\nWords:\n48,204\nChapters:\n18/?\nComments:\n55\nKudos:\n94\nBookmarks:\n5\nHits:\n4444\n[GGAD/ADGG] Try and Stop Me by do not cache AsphodelV\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nMature\nChoose Not To Use Archive Warnings\nM/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nAlbus Dumbledore/Gellert Grindelwald\nAlbus Dumbledore & Gellert Grindelwald\nAlbus Dumbledore\nGellert Grindelwald\nNewt Scamander\nVinda Rosier\nAberforth Dumbledore\nTime Travel\nTime Travel Fix-It\nAlternate Universe - Time Travel\nPOV Gellert Grindelwald\nSuicidal Thoughts\nSummary\n\nGellert Grindelwald travels back to the duel of 1945, and gets urges to kill himself. Hopefully, Albus can help with that.\n\nLanguage:\nEnglish\nWords:\n8,338\nChapters:\n5/?\nComments:\n31\nKudos:\n163\nBookmarks:\n15\nHits:\n2139\nThough We Cannot Make Our Sun Stand Still by do not cache Herbalspells (Jedi_librarian)\nFandoms: Fantastic Beasts and Where to Find Them (Movies), Harry Potter - J. K. Rowling \nMature\nNo Archive Warnings Apply\nM/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nNo Archive Warnings Apply\nAlbus Dumbledore/Gellert Grindelwald\nAlbus Dumbledore & Newt Scamander\nAberforth Dumbledore & Albus Dumbledore\nAlbus Dumbledore & Minerva McGonagall\nElphias Doge & Albus Dumbledore\nAlbus Dumbledore\nGellert Grindelwald\nNewt Scamander\nTheseus Scamander\nJacob Kowalski\nQueenie Goldstein\nTina Goldstein\nAberforth Dumbledore\nCredence Barebone\nAurelius Dumbledore\nMinerva McGonagall\nEulalie Hicks\nAriana Dumbledore\nPercival Dumbledore\nKendra Dumbledore\nNicolas Flamel (Harry Potter)\nElphias Doge\nYoung Albus Dumbledore\nPost Secrets of Dumbledore\nMultiverse\nTime Travel\nAlternate Universe - Canon Divergence\nNurmengard Castle (Harry Potter)\nHogwarts\nGodric's Hollow (Harry Potter)\nYoung Gellert Grindelwald\nDurmstrang\nPeriod-Typical Homophobia\nPeriod-Typical Sexism\nNazi Germany\nIdiots in Love\nFix-It\nHistorical References\nHistorical Figures\nPeriod Typical Attitudes\nSummary\n\nPost-Secrets of Dumbledore AU.\n\n**Spoiler warning**\nPlease don’t read if you haven’t watched Secrets of Dumbledore yet and want to remain spoiler-free :)\n\n***\n\nAlbus sits in his office, conflicted, remembering his emotional duel with Gellert. He decides that if there’s any possibility of saving the man he loves - without risking the safety of the Muggle world - then he has to try. After all, isn’t love meant to be as strong as the most powerful magic?\n\nSeries\nPart 1 of Though We Cannot Make Our Sun Stand Still\nLanguage:\nEnglish\nWords:\n101,027\nChapters:\n13/21\nCollections:\n1\nComments:\n261\nKudos:\n550\nBookmarks:\n67\nHits:\n11197\nFor Reasons Wretched and Divine by do not cache illicitopia\nFandoms: Fantastic Beasts and Where to Find Them (Movies) \nTeen And Up Audiences\nNo Archive Warnings Apply\nM/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nNo Archive Warnings Apply\nAlbus Dumbledore/Gellert Grindelwald\nAlbus Dumbledore\nGellert Grindelwald\nMinerva McGonagall\nNewt Scamander\nVinda Rosier\nQueenie Goldstein\nLight Angst\nGrindeldore\nEnemies to Lovers\nMutual Pining\nNot Canon Compliant\nGellert isn't evil\nMirror of Erised au\nAlbus Dumbledore Needs a Hug\nVinda is a good wingman\nhow do I even tag this stuff\ncan be taken as minor Vinda/Queenie if you want\nit's fine with me\nMinerva is done with Albus's bullshit\nMild Hurt/Comfort\nAngst and Feels\nSummary\n\nAlbus Dumbledore and Gellert Grindelwald are left in a whirlwind of emotions after Albus leaves Gellert to his plot for the greater good. Neither quite knows what to do with themselves after what happened, but a series of unexpected events seems to thrust them right into each other. Can they somehow manage to heal the wounds they gave each other, or will they leave with a far greater brokenness than they even knew possible?\n\nLanguage:\nEnglish\nWords:\n3,254\nChapters:\n3/?\nComments:\n6\nKudos:\n18\nBookmarks:\n3\nHits:\n192\nDance with me by do not cache Metallica1981\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nGeneral Audiences\nNo Archive Warnings Apply, Underage\nM/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nNo Archive Warnings Apply\nUnderage\nAlbus Dumbledore/Gellert Grindelwald\nAlbus Dumbledore\nGellert Grindelwald\nElphias Doge\nOriginal Characters\nArmando Dippet\nPhineas Nigellus Black\nNot Canon Compliant\nAU Triwizard Tournament\nFriends to Lovers\nGetting Together\nYule Ball\nInspired by Tumblr\nPre-Movie 1: Fantastic Beasts and Where to Find Them\nSmoking\nUnderage Smoking\nCanon-Typical Violence\nSwearing\nSummary\n\nAlbus Dumbledore is a student at Hogwarts when the Triwizard Tournament is taking place at Hogwarts - when he meets Gellert Grindelwald, a Durmstrang student sent to Hogwarts as part of the tournament.\n\nLanguage:\nEnglish\nWords:\n22,554\nChapters:\n11/?\nComments:\n98\nKudos:\n133\nBookmarks:\n11\nHits:\n1539\nUncle Percy by do not cache Umeko\nFandoms: Fantastic Beasts and Where to Find Them (Movies) \nTeen And Up Audiences\nChoose Not To Use Archive Warnings\nF/M\nWork in Progress\n\n11 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nTina Goldstein/Newt Scamander\nTina Goldstein & Original Percival Graves\nQueenie Goldstein/Jacob Kowalski\nQueenie Goldstein & Original Percival Graves\nQueenie Goldstein & Tina Goldstein\nOriginal Percival Graves & Seraphina Picquery\nOriginal Percival Graves/Seraphina Picquery\nTina Goldstein\nQueenie Goldstein\nOriginal Percival Graves\nJacob Kowalski\nNewt Scamander\nTheseus Scamander\nSeraphina Picquery\nOriginal Characters\nMacDuff (Fantastic Beasts)\nMACUSA Staff\nGellert Grindelwald\nOriginal Percival Graves' Mother\nOriginal Percival Graves' Father\nOriginal House-Elf Character(s)\nFamily Drama\nAngst\nImplied/Referenced Torture\nMental Institutions\nMental Health Issues\nPhysical Disability\nRehabilitation\nImplied/Referenced Abuse\nDysfunctional Family\nFriendship\nLoss\nRomance\nSuicide Attempt\nSummary\n\nPercival Graves did not expect that accepting his mentor's dinner invitation would lead to a long-lasting friendship with two sisters.\n\nLanguage:\nEnglish\nWords:\n12,177\nChapters:\n6/?\nComments:\n19\nKudos:\n20\nBookmarks:\n1\nHits:\n267\nlost in translation - tmr by do not cache jqtunheims\nFandoms: Harry Potter - J. K. Rowling, Fantastic Beasts and Where to Find Them (Movies) \nMature\nChoose Not To Use Archive Warnings, Graphic Depictions Of Violence\nMulti\nWork in Progress\n\n11 Sep 2022\n\nTags\nCreator Chose Not To Use Archive Warnings\nGraphic Depictions Of Violence\nTom Riddle/Original Female Character(s)\nTom Riddle\nOriginal Lestrange Characters (Harry Potter)\nAlbus Dumbledore\nAbraxas Malfoy\nOlive Hornby\nDruella Rosier Black\nOriginal Nott Family Characters (Harry Potter)\nOriginal Hogwarts Professors\nOriginal House-Elf Character(s)\nOriginal Female Character(s)\nRosier Family (Harry Potter)\nGellert Grindelwald\nVinda Rosier\nEveryone Is Gay\nPost-Traumatic Stress Disorder - PTSD\nBorderline Personality Disorder\nDead Dove: Do Not Eat\nEnemies to Lovers\nHogwarts\nRivals With Benefits\nOther Additional Tags to Be Added\nMental Health Issues\nThe Deathly Hallows\nKnights of Walpurgis (Harry Potter)\nEventual Smut\nInspired by Taylor Swift\nNot Canon Compliant - Movie 2: Fantastic Beasts: The Crimes of Grindelwald\nFantasy\nDark Romance\nLGBTQ Character\nLana Del Rey References\nSong: All Too Well (Taylor Swift)\nSummary\n\nthe age old star crossed lovers story, the most famous of all being romeo and juliet. this? this is worse.\n\narabella was born to do great things, so she was told. upon being sent to hogwarts after being educated privately, she's thrust into a world she doesn't know how to navigate. mysteries upon mysteries and friendships mirrored with betrayal, arabella is left lonely and confused.\n\nso, enter her \"savior.\"\n\nright??\n\nLanguage:\nEnglish\nWords:\n11,703\nChapters:\n9/?\nKudos:\n3\nBookmarks:\n1\nHits:\n109\n[NSFW Arts] Jackie Kowalski and Artemis Scamander by do not cache your_old_PC\nFandoms: Fantastic Beasts and Where to Find Them (Movies) \nExplicit\nNo Archive Warnings Apply\nF/F\nWork in Progress\n\n11 Sep 2022\n\nTags\nNo Archive Warnings Apply\nfem!Jacob Kowalski/fem!Newt Scamander\nJacob Kowalski/Newt Scamander\nfem!Jacob Kowalski - Character\nfem!Newt Scamander - Character\nJacob Kowalski\nNewt Scamander\nDigital Art\nNSFW Art\nGenderbending\nBDSM\nCollars\nSex Toys\nNudity\nWeddings\nSummary\n\nVarious arts with Jackie and Artemis (fem!newcob)\n\nLanguage:\nEnglish\nWords:\n0\nChapters:\n5/?\nComments:\n7\nKudos:\n101\nBookmarks:\n4\nHits:\n5449\nFilters\nFilter results:\nSubmit\nSort by\nAuthorTitleDate PostedDate UpdatedWord CountHitsKudosCommentsBookmarks\nInclude\n?\nInclude Ratings\nGeneral Audiences (4604)\nTeen And Up Audiences (4131)\nExplicit (2711)\nMature (2551)\nNot Rated (1649)\nInclude Warnings\nNo Archive Warnings Apply (7347)\nCreator Chose Not To Use Archive Warnings (6587)\nGraphic Depictions Of Violence (1098)\nRape/Non-Con (646)\nMajor Character Death (640)\nUnderage (470)\nInclude Categories\nM/M (9440)\nF/M (3943)\nGen (2535)\nMulti (561)\nF/F (417)\nOther (408)\nInclude Fandoms\nFantastic Beasts and Where to Find Them (Movies) (15646)\nHarry Potter - J. K. Rowling (3964)\nFantastic Beasts and Where to Find Them (Movies) RPF (71)\nMarvel Cinematic Universe (67)\nHarry Potter and the Cursed Child - Thorne & Rowling (54)\nThe Avengers (Marvel Movies) (39)\nBritish Actor RPF (38)\nElla Enchanted (2004) (37)\nSherlock (TV) (32)\nFright Night (2011) (28)\nInclude Characters\nNewt Scamander (8000)\nOriginal Percival Graves (5063)\nTina Goldstein (4479)\nGellert Grindelwald (4037)\nCredence Barebone (3672)\nAlbus Dumbledore (3507)\nTheseus Scamander (3313)\nQueenie Goldstein (3073)\nJacob Kowalski (2121)\nSeraphina Picquery (1124)\nInclude Relationships\nAlbus Dumbledore/Gellert Grindelwald (2779)\nTina Goldstein/Newt Scamander (2143)\nOriginal Percival Graves/Newt Scamander (1952)\nCredence Barebone/Original Percival Graves (1751)\nQueenie Goldstein/Jacob Kowalski (1092)\nNewt Scamander/Theseus Scamander (1031)\nNewt Scamander & Theseus Scamander (721)\nTina Goldstein & Newt Scamander (638)\nCredence Barebone/Newt Scamander (501)\nCredence Barebone/Percival Graves | Gellert Grindelwald (481)\nInclude Additional Tags\nFluff (2185)\nAngst (1406)\nHurt/Comfort (1149)\nAlternate Universe - Canon Divergence (910)\nRomance (884)\nAlternate Universe (755)\nAlternate Universe - Modern Setting (535)\nDrama (525)\nFluff and Angst (522)\nPlot What Plot/Porn Without Plot (505)\nOther tags to include\nExclude\n?\nExclude Ratings\nGeneral Audiences (4604)\nTeen And Up Audiences (4131)\nExplicit (2711)\nMature (2551)\nNot Rated (1649)\nExclude Warnings\nNo Archive Warnings Apply (7347)\nCreator Chose Not To Use Archive Warnings (6587)\nGraphic Depictions Of Violence (1098)\nRape/Non-Con (646)\nMajor Character Death (640)\nUnderage (470)\nExclude Categories\nM/M (9440)\nF/M (3943)\nGen (2535)\nMulti (561)\nF/F (417)\nOther (408)\nExclude Fandoms\nFantastic Beasts and Where to Find Them (Movies) (15646)\nHarry Potter - J. K. Rowling (3964)\nFantastic Beasts and Where to Find Them (Movies) RPF (71)\nMarvel Cinematic Universe (67)\nHarry Potter and the Cursed Child - Thorne & Rowling (54)\nThe Avengers (Marvel Movies) (39)\nBritish Actor RPF (38)\nElla Enchanted (2004) (37)\nSherlock (TV) (32)\nFright Night (2011) (28)\nExclude Characters\nNewt Scamander (8000)\nOriginal Percival Graves (5063)\nTina Goldstein (4479)\nGellert Grindelwald (4037)\nCredence Barebone (3672)\nAlbus Dumbledore (3507)\nTheseus Scamander (3313)\nQueenie Goldstein (3073)\nJacob Kowalski (2121)\nSeraphina Picquery (1124)\nExclude Relationships\nAlbus Dumbledore/Gellert Grindelwald (2779)\nTina Goldstein/Newt Scamander (2143)\nOriginal Percival Graves/Newt Scamander (1952)\nCredence Barebone/Original Percival Graves (1751)\nQueenie Goldstein/Jacob Kowalski (1092)\nNewt Scamander/Theseus Scamander (1031)\nNewt Scamander & Theseus Scamander (721)\nTina Goldstein & Newt Scamander (638)\nCredence Barebone/Newt Scamander (501)\nCredence Barebone/Percival Graves | Gellert Grindelwald (481)\nExclude Additional Tags\nFluff (2185)\nAngst (1406)\nHurt/Comfort (1149)\nAlternate Universe - Canon Divergence (910)\nRomance (884)\nAlternate Universe (755)\nAlternate Universe - Modern Setting (535)\nDrama (525)\nFluff and Angst (522)\nPlot What Plot/Porn Without Plot (505)\nOther tags to exclude\nMore Options\nCrossovers\nInclude crossovers\nExclude crossovers\nShow only crossovers\nCompletion Status\nAll works\nComplete works only\nWorks in progress only\nWord Count\nFrom\nTo\nDate Updated\nFrom\nTo\nSearch within results ?\nLanguage\naf SoomaaliAfrikaansالعربية𓂋𓏺𓈖 𓆎𓅓𓏏𓊖ܐܪܡܝܐ | ארמיאհայերենasturianuBahasa IndonesiaBahasa MalaysiaБългарскиবাংলাBasa JawaБашҡорт телебеларускаяBosanskiBrezhonegCatalàCebuanoČeštinaChinuk WawaCymraegDanskDeutscheesti keelΕλληνικά𒅴𒂠EnglishEald EnglisċEspañolEsperantoEuskaraفارسیFilipinoFrançaisFurlanGaeilgeGàidhligGalego𐌲𐌿𐍄𐌹𐍃𐌺𐌰中文-客家话한국어Hausa | هَرْشَن هَوْسَहिन्दीHrvatskiʻŌlelo HawaiʻiInterlinguaisiZuluÍslenskaItalianoעבריתಕನ್ನಡქართულიភាសាខ្មែរKhuzdulKiswahilikreyòl ayisyenKurdî | کوردیКыргызчаLangue des signes québécoiseLatviešu valodaLëtzebuergeschLietuvių kalbaLingua latinaMagyarмакедонскиമലയാളംMaltiᠮᠠᠨᠵᡠ ᡤᡳᠰᡠᠨमराठीMikisúkîᠮᠣᠩᠭᠣᠯ ᠪᠢᠴᠢᠭ᠌ | Монгол Кирилл үсэгမြန်မာဘာသာNāhuatl中文-闽南话 臺語Nederlands日本語NorskАзәрбајҹан дили | آذربایجان دیلیНохчийн моттلسان عثمانىپښتوPlattdüütschPolskiPortuguês brasileiroPortuguês europeuਪੰਜਾਬੀqazaqşa | қазақшаQuenyaRomânăРусскийScotsShqipSindarinසිංහලSlovenčinaSlovenščinaSprēkō ÞiudiskōСрпскиSuomiSvenskaதமிழ்తెలుగుไทยThermianབོད་སྐད་Tiếng Việtϯⲙⲉⲧⲣⲉⲙⲛ̀ⲭⲏⲙⲓtlhIngan-Holtoki ponaτσακώνικαTürkçeУкраїнськаاُردُوئۇيغۇر تىلىVolapük中文-吴语יידישmaayaʼ tʼàan中文-广东话 粵語中文-普通话 國語\nSubmit\n\nClear Filters\n\nTop of Work Index\n\nPages Navigation\n← Previous\n1\n2\n3\n4\n5\n6\n7\n8\n9\n…\n782\n783\nNext →\npresentational\nFooter\nAbout the Archive\nSite Map\nDiversity Statement\nTerms of Service\nDMCA Policy\nContact Us\nPolicy Questions & Abuse Reports\nTechnical Support & Feedback\nDevelopment\notwarchive v0.9.324.8\nKnown Issues\nGPL by the OTW\nif(typeof jQuery=='undefined'){document.write(unescape(\"%3Cscript src='/javascripts/jquery.min.js' type='text/javascript'%3E%3C/script%3E\"));document.write(unescape(\"%3Cscript src='/javascripts/jquery-ui.min.js' type='text/javascript'%3E%3C/script%3E\"));}$j=jQuery.noConflict();eval(mod_pagespeed_a8hDjzT3l_);eval(mod_pagespeed_1nA1vKz9z4);eval(mod_pagespeed_AKC4FLip6q);eval(mod_pagespeed_gX7XC77lvT);eval(mod_pagespeed_5zo0ZWMy4O);eval(mod_pagespeed_eCCgOhjhYQ);eval(mod_pagespeed_TU1Ln7LKSh);eval(mod_pagespeed_qTpbjsXQwh);eval(mod_pagespeed_R5FGw2ySDb);eval(mod_pagespeed_rqqFCT3xcT);eval(mod_pagespeed_Gf_4KFhTvB);function isSupported(){var item=\"localStoragePolyfill\";try{localStorage.setItem(item,item);localStorage.removeItem(item);return true;}catch(e){return false;}}function acceptTOS(){if(isSupported()){localStorage.setItem(\"accepted_tos\",\"20180523\");}else{Cookies.set(\"accepted_tos\",\"20180523\",{expires:365});}}$j(document).ready(function(){if(localStorage.getItem(\"accepted_tos\")!==\"20180523\"&&Cookies.get(\"accepted_tos\")!==\"20180523\"){$j(\"body\").prepend(\"<div id=\\\"tos_prompt\\\" class=\\\"hidden\\\">\\n <h2 class=\\\"heading\\\">\\n <span>Archive of Our Own<\\/span>\\n <\\/h2>\\n <div class=\\\"agreement\\\">\\n <p class=\\\"summary\\\">\\n On Archive of Our Own (AO3), users can make profiles, create works and\\n other Content, post comments, give Kudos, create Collections and \\n Bookmarks, participate in Challenges, import works, and more. Any \\n information you publish in a comment, profile, work, or Content that you \\n post or import onto AO3 including in summaries, notes and tags, \\n will be accessible by the public (unless you limit access to a work only \\n to those with AO3 Accounts), and it will be available to \\n AO3 personnel. Be mindful when sharing personal information, \\n including your religious or political views, health, racial background, \\n country of origin, sexual identity and/or personal relationships. To \\n learn more, check out our <a href=\\\"/tos\\\">Terms of Service<\\/a> and <a href=\\\"/tos#privacy\\\">Privacy Policy<\\/a>.\\n <\\/p>\\n\\n <p class=\\\"confirmation\\\">\\n <input type=\\\"checkbox\\\" id=\\\"tos_agree\\\" />\\n <label for=\\\"tos_agree\\\">I have read & understood the new Terms of Service and Privacy Policy<\\/label>\\n <\\/p>\\n\\n <p class=\\\"submit\\\">\\n <button name=\\\"button\\\" type=\\\"button\\\" disabled=\\\"disabled\\\" id=\\\"accept_tos\\\">I agree/consent to its terms<\\/button>\\n <\\/p>\\n\\n <\\/div>\\n<\\/div>\\n\\n<script>\\n//<![CDATA[\\n\\n \\$j(document).ready(function() {\\n var container = \\$j(\\\"#tos_prompt\\\");\\n var outer = \\$j(\\\"#outer\\\");\\n var button = \\$j(\\\"#accept_tos\\\");\\n\\n setTimeout(showTOSPrompt, 1500);\\n\\n function showTOSPrompt() {\\n \\$j.when(container.fadeIn(500)).done(function() {\\n outer.addClass(\\\"hidden\\\").attr(\\\"aria-hidden\\\", \\\"true\\\");\\n });\\n\\n \\$j(\\\"#tos_agree\\\").on(\\\"click\\\", function() {\\n button.attr(\\\"disabled\\\", !this.checked);\\n if (this.checked) {\\n button.on(\\\"click\\\", function() {\\n acceptTOS();\\n outer.removeClass(\\\"hidden\\\").removeAttr(\\\"aria-hidden\\\");\\n \\$j.when(container.fadeOut(500)).done(function() {\\n container.remove();\\n });\\n });\\n };\\n }).change();\\n };\\n });\\n\\n//]]]]><![CDATA[>\\n<\\/script>\");}});$j(document).ready(function(){var permitted_hosts=[\"104.153.64.122\",\"208.85.241.152\",\"208.85.241.157\",\"archiveofourown.org\",\"download.archiveofourown.org\",\"insecure.archiveofourown.org\",\"secure.archiveofourown.org\",\"www.archiveofourown.com\",\"www.archiveofourown.net\",\"www.archiveofourown.org\",\"insecure-test.archiveofourown.org\",\"test.archiveofourown.org\",\"testdownload.archiveofourown.org\"];var current_host=window.location.hostname;if(!permitted_hosts.includes(current_host)&&Cookies.get(\"proxy_notice\")!==\"0\"&&window.location.protocol!==\"file:\"){$j(\"#skiplinks\").after(\"<div id=\\\"proxy-notice\\\">\\n <div class=\\\"userstuff\\\">\\n <p class=\\\"important\\\">Important message:<\\/p>\\n <ol>\\n <li>You are using a proxy site that is not part of the Archive of Our Own.<\\/li>\\n <li>The entity that set up the proxy site can see what you submit, including your IP address. If you log in through the proxy site, it can see your password.<\\/li>\\n <\\/ol>\\n <p class=\\\"important\\\">重要提示：<\\/p>\\n <ol>\\n <li>您使用的是第三方开发的反向代理网站，此网站并非Archive of Our Own - AO3（AO3作品库）原站。<\\/li>\\n <li>代理网站的开发者能够获取您上传至该站点的全部内容，包括您的ip地址。如您通过代理登录AO3，对方将获得您的密码。<\\/li>\\n <\\/ol>\\n <p class=\\\"submit\\\"><button class=\\\"action\\\" type=\\\"button\\\" id=\\\"proxy-notice-dismiss\\\">Dismiss Notice<\\/button><\\/p>\\n <\\/div>\\n<\\/div>\\n\\n<script>\\n//<![CDATA[\\n\\n \\$j(document).ready(function() {\\n \\$j(\\\"#proxy-notice-dismiss\\\").on(\\\"click\\\", function() {\\n Cookies.set(\\\"proxy_notice\\\", \\\"0\\\");\\n \\$j(\\\"#proxy-notice\\\").slideUp();\\n });\\n });\\n\\n//]]]]><![CDATA[>\\n<\\/script>\");}});$j(\".datepicker\").datepicker({dateFormat:\"yy-mm-dd\"});"Fanfiction Study: Scraper
Build Function
get_pageworkIDs <- function(input){
input |>
html_elements(css = "ol.work.index.group") |>
as.character() |>
str_match_all("<li id=(.*?) class=") |>
data.frame() |>
select("X2") |>
mutate(X2 = gsub('[[:punct:]]', "", X2),
X2 = gsub("work", "", X2)) |>
rename(id = X2) |>
mutate(id = as.numeric(id),
id = paste0("https://archiveofourown.org/works/", id))
}Fanfiction Study: Scraper
Extract Information
input <- get_html(input = "https://archiveofourown.org/tags/Fantastic%20Beasts%20and%20Where%20to%20Find%20Them%20(Movies)/works")
get_pageworkIDs(input) id
1 https://archiveofourown.org/works/41272776
2 https://archiveofourown.org/works/38394940
3 https://archiveofourown.org/works/35525524
4 https://archiveofourown.org/works/35206012
5 https://archiveofourown.org/works/40409019
6 https://archiveofourown.org/works/41462481
7 https://archiveofourown.org/works/41221239
8 https://archiveofourown.org/works/39508659
9 https://archiveofourown.org/works/41652618
10 https://archiveofourown.org/works/41650074
11 https://archiveofourown.org/works/41648583
12 https://archiveofourown.org/works/41648052
13 https://archiveofourown.org/works/38286550
14 https://archiveofourown.org/works/36514978
15 https://archiveofourown.org/works/38290693
16 https://archiveofourown.org/works/41596014
17 https://archiveofourown.org/works/41340351
18 https://archiveofourown.org/works/41465799
19 https://archiveofourown.org/works/41364669
20 https://archiveofourown.org/works/26163109Fanfiction Study: Scraper
Fanfiction Study: Scraper
GetFandomIndex <- function(fandom, pages, start = 1) {
### Get number of pages
indexURL <- get_indexURL(fandom)
index <- get_html(indexURL)
indexpages <- get_indexpages(index)
pagecount <- min(pages, indexpages)
index_ids <- as.data.frame(matrix(ncol = 1,
nrow = 0))
### Scrape each page, store in dataframe
for (i in start:pagecount) {
pageURL <- get_indexURLpage(indexURL, i)
page_html <- get_html(pageURL)
id_list <- get_pageworkIDs(page_html)
index_ids <- rbind(index_ids, id_list)
}
### Output dataframe with all fanfictions
index_ids
}Fanfiction Study: Scraper
References
Research Methods: Webscraping